Rate Distortion Optimization in H.264

In this paper, we study the best rate distortion performance that an H.264 encoder can possibly achieve. Using soft decision quantization rather than the standard hard decision quantization, we first establish a general framework for jointly designing motion compensation, quantization, and entropy coding in the hybrid coding structure of H.264 to minimize a true rate distortion cost. We then propose three rate distortion optimization algorithms—a graph-based algorithm for optimal soft decision quantization in H.264 baseline profile encoding given motion compensation and quantization step sizes, an iterative algorithm for optimal residual coding in H.264 baseline profile encoding given motion compensation, and an iterative overall algorithm for optimal H.264 baseline profile encoding—with them embedded in the indicated order. The graph-based algorithm for optimal soft decision quantization is the core; given motion compensation and quantization step sizes, it is guaranteed to perform optimal soft decision quantization to certain degree. The proposed iterative overall algorithm has been implemented based on the reference encoder JM82 of H.264. Comparative studies show that it achieves a significant performance gain, which can be as high as 25% rate reduction at the same PSNR when compared to the reference encoder.